Application Serial No. 09/757,930 Page 2 of 10 

Date April 7, 2004 

Reply to Office Action dated January 7, 2004 

Listing of the Claims: ^ 

1 . (Currently Amended) A system for interacting with displays and 
all devices that use such displays comprised of 

a. a display, 

b. a s e nsor or cam e ra a pointing object, 

c. a pointing d e vic e r e mot e from th e s e nsor or a camera and that 
has in its field of view at least one of the following: (i) the pointing object only, or 
Cn) the display and the pointing object, or Cm) the display and the reflection or effect 
that the pointing object can produce on the display, can b e viewed by the s e nsor or 
cam e ra, wh e r e in th e sensor or cam e ra op e rat e s ind e p e nd e ntly from the pointing 
d e vice, 

d. a method for detecting the position of the pointing d e vice, 
object or its reflection or effect on the display in the image registered by the camera, 
and 

e. a method for establishing the mapping between the position of 
the pointing d e vic e object or its reflection or effect on the display in the image 
registered bv the camera and a corresponding location on the display. 



2. (Currently Amended) A system according to as defined bv 
claim 1 wh e r e in th e s e nsor or camera, in addition to vi e wing th e imag e of th e 
pointing obj e ct, can also view at least on e of (i) th e image of th e display, (ii) the 
reflection or e f fect that the pointing devic e can produce on the display and (iii) space 
around the pointing d e vic e which commands the positioning of a pointing icon on the 
display. 



3. (Currently Amended) A system as defined by claim 1 which 
commands the positioning of a pointing icon on input of data into the device using 
the display. 
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4. (Currently Amended) A system according to claim 1 wherein 
the pointing device object is a part of the human body of a user of the system such as 
tt the hand or a the finger of the user , or an ornament or an object or device worn on 
the human body of the user, such as a glove or thimble and wherein the sensor or 
camera is remote from the human body pointing object . 

5. (Currently Amended) A system according to claim 4 wherein 
the pointing devic e object is used to point to regions of the display by way of 
changing its position, attitude, or presentation. 

6. (Currently Amended) A system according to claim 1 wherein 
the pointing d e vic e object is used to select highlight, or define a particular point or 
region on the display. 

1. (Currently Amended) A system according to claim 1 wherein 
the pointing device object is used to define a vector on the plane of the display that 
indicates a direction and magnitude relative to or with respect to an item on the 
display or a region of the display. 

8. (Currently Amended) A system according to claim 3- 2 wherein 
the pointing icon on the display can be registered by the s e nsor or camera. 

9. (Currently Amended) A system according to claim 8 which also 
includes a method for correcting the offsets between (i) the position of the pointing 
device object , or reflection, or effect thereof on the display as observed by the user or 
by the s e nsor o r camera, and (ii) the position of the pointer icon on the display. 



10. (Original) A system as defined by claim 1 which also includes 
at least one of the following: 
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a. a method for selecting or highlighting a specific item or icon on 

the display, 

b. a method for activating a specific process, program, or menu 
item represented on the display, and 

c. a method for writing, scribing, drawing, highlighting, 
annotating, or otherwise producing marks on the display. 

1 1 . (Currently Amended) A method for detecting a pointing d e vice 
position of the pointing object or its reflection or effect on the display in the image 
registered by a camera comprising the steps of: 

1L defining at least one characteristic of the pointing object or its 
reflection or effect on the display that (i) is registered in the image captured by the 
camera and (ii) distinguishes the pointing object from other objects registered in the 
image from the camera. 

nb. retrieving of data or the image from a sensor or the camera, 
wh e r e in th e pointing d e vice is remote from th e s e nsor or cam e ra, and 

be. analyzing the data or image from the s e nsor or camera to locate 
the characteristic or characteristics of the pointing d e vic e object in the data, or 
locating at least a set of the pictur e el e m e nts in th e image that compris e th e r e ndition 
of th e its reflection or effect on the display pointing d e vic e object [.] 

d determining the most likely position of the pointing object or its 
reflection or effect on the display in the image from the camera based on at least one 
of the following: 

(i) the last known position of the pointing object or its reflection 
or effect on the display in the image, 

(ii) the position or positions at which the at least one 
distinguishing characteristic of the pointing object, or its reflection or effect on the 
display or the set of the picture elements in the image that comprise the rendition of 
the pointing object, or its reflection or effect on the display. 
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12. (Currently Amended) A method according to claim 1 1 wherein 
characteristics at least one characteristic that distinguish distinguishes the pointing 
d e vic e object from other objects in the data image from the sensor or th e image from 
the camera are is known a priori. 

13. (Currently Amended) A method according to claim 1 1 wherein 
charact e ristics at least one characteristic that distinguish distinguishes the pointing 
devic e object from other objects in the data from th e s e nsor or the image from the 
camera are determined based on analysis of at least one set of the data acquired from 
the sensor or one image acquired from the camera. 

14. (Currently Amended) A method according to claim 13 wherein 
th e charact e ristics at least one characteristic that distinguish distinguishes the 
pointing d e vic e object from other objects, whose rendition are present in th e data 
from th e s e nsor or in the image from the camera, is obtained by 

a. acquiring at least two s e ts of data from th e sensor or images 
from the camera, one with the pointing d e vic e object in view of the s e nsor 

or the camera and one without, and 

b. comparing the two sets with one another. 

15. (Currently Amended) A method according to claim 1 1 wherein 
adjustments or modifications are made to the position, viewing angles, sensitivity, 
and other settings of the sens o r or the camera pursuant the analysis of the data or 
image retrieved from the sensor or the camera. 

16. (Currently Amended) A method according to claim 1 1 wherein 
at least part of the procedures for the method is carried out using at least in part the 
computing mechanisms available on one or more of the following: the display, or th e 
sensor or the camera, or the pointing device, or the device producing the signal 
shown on the display, or the device producing the pointing icon on the display. 
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17. (Currently Amended) A method for establishing a mapping 
between the set of positions that a pointing d e vic e object can assume in addressing a 
set of corresponding points or regions on the display comprising the steps of: 

a T defining the rang e of positions that th e pointing devic e can 
assum e , boundaries of the positions that the pointing object can assume in addressing 
points or regions on the display and defining, within the said boundaries, a 
continuous pointing object surface. 

b. defining the boundaries of the positi o ns that the pointing d e vice 
can assum e so that a virtual display space comprising (i) a continuous one- 
dim e nsional line, or (ii) a continuous two-dim e nsional plan e , or (iii) a continuous 
three-dimensional volume is defined display and defining within the boundaries of 
the display, a continuous pointing object surface . 

c. defining th e boundari e s o f segmenting the display so that a r e al 
display space comprising (i) a continuous on e -dim e nsional lin e , or (ii) a continuous 
two-dim e nsional plane, or (iii) a continuous thr ee -dim e nsional volum e is d e fin e d, 
and surface into at least two regions. 

d. warping th e g e om e try of th e r e al display spac e so that the r e al 
display spac e fits optimally within the boundaries of the virtual display spac e 
segmenting the pointing object surface into at least two regions. 

e^ warping the geometry of the surface for the pointing ob ject so 
that at least one region of the pointing object surface overlaps with at least one region 
of the display surface, and 

f establishing a one-to-one or many to one correspondence 
between overlapping the regions of the pointing object surface and the display 
surface, respectively. 

18. (Currently Amended) A method according to claim 17 wherein 
the boundaries of the set of positions that the pointing device object can assume are 
obtained by querying the user to point to the boundaries. 
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19 (Currently Amended) A method according to claim 17 wherein 
the boundaries of the set of positions that the pointing device ob ject can assume are 
obtained by contours or the periphery of the display surface as the display surface is 
vi e w e d present in the image from by th e s e nsor or the camera. 

20. (Currently Amended) A method according to claim -r9 17 
wherein at least on e sp e cial display imag e is used for e stablishing the mapping 
betw ee n th e s e t of positions that th e pointing device can assum e in addr e ssing a s e t of 
corresponding points the regions of the display surface comprise at least two sets of 
pixel elements that comprise the image on the display. 

21. (Currently Amended) A method according to claim 17 wherein 
at least part of the procedures for the method is carried out using at least in part the 
computing mechanisms available on one or more of the following: the display, or the 
s e nsor or camera, or the pointing device object or the device producing the signal 
shown on the display, or the device producing the pointing icon on the display. 

Cancel claims 22-28. 



29. (New) A method for establishing a mapping between the 
regions of a display and an image of the same display obtained from a camera 
comprising the steps of: 

a. defining the boundaries of the display image, 

b. defining within the boundaries of the display image, a 
continuous display surface, 

c. segmenting the display surface into at least two regions, 

d. segmenting the display into at least two regions, 

e. warping the geometry of the display image surface so that at 
least one region of the display image surface overlaps with at least one region of the 
display surface, and 
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f. establishing a one-to-one or many to one correspondence 
between overlapping the regions of the display image surface and the display surface, 
respectively. 

30. (New) A method according to claim 29 wherein at least one 
special image is rendered on the display for establishing a mapping between the 
regions of a display and an image of the same display obtained from a camera. 



